Method for pupil detection for cognitive monitoring, analysis, and biofeedback-based treatment and training

ABSTRACT

The present invention relates to a system and method for pupil tracking and eye-markers extracting using an image acquisition device such as a visible-light camera shooting a non-static head. In an embodiment of the present invention eye-markers are extracted from one eye or from both eyes. The extraction from both eyes uses for averaging the results of the two eyes when abnormalities are detected in one of the eyes. In addition, the invention relates to a computerized application, which interacts with a user (for example trough a game or movie, usually in a context of biofeedback), and takes video shooting of the users face and eyes during said interaction.

FIELD OF THE INVENTION

The invention relates to the field of human cognitive monitoring througheye measures analysis using computer vision and image processing. Morespecifically, the present invention relates to a method for pupildetection using a standard digital image acquisition device forcognitive monitoring, analysis and biofeedback sensor based treatmentand training.

BACKGROUND OF THE INVENTION

Cognitive performance can be monitored and trained using physiologicalmeasures. Like other competencies, cognitive capabilities stem from acombination of “nature vs nurture” parameters, namely those inheritedvia genetics and those related to learning and development via exposureto the environment. Cognitive improvement solutions using biofeedbackare common in many fields, starting with the treatment of people withdisorders like ADHD, PDD and head injuries, to people who seek toimprove their cognitive abilities (e.g., athletes, students ormanagers).

Biofeedback is a closed-loop process of using a bio-sensor (measuringone or more biometric markers) to modify (normally improve) userperformance/condition/behavior by providing the user with feedbackpromoting a desired change.

Combining the high efficiency and transferability of biofeedbacksolutions with the ubiquity of computer and mobile applications, maysupply a good solution for cognitive enhancement.

Recent research indicates that changes in physiological andpsychological state are reflected in eye measures (such as, but notlimited to pupil diameter, saccades and blinks). However, standard meansof analyzing eye measures require the use of very expensive equipment,such as special IR cameras which are not commonly accessible. Operationof such equipment typically requires professional knowhow, thus makingit out of reach for daily usage by the general public.

Therefore, a solution is needed, which uses a standard digital imageacquisition device for this exact purpose. To the best of the inventorsknowledge such attempt to monitor the eyes and pupil by use of regularequipment (e.g., a video camera in a standard smartphone) has not yetbeen accomplished. A main challenge when attempting to analyze eyes datafor such cognition related purposes is the need for highly accurateextraction of markers (such as pupil diameter, pupil center location,blinks and their changes over time) from a video stream or file.

Other challenges that need to be addressed when extracting pupil and eyemetrics using standard digital image acquisition device (such as in asmartphone front cameras) are: adjusting to limited quality camera,dynamic lighting conditions (brightness, backlight, reflections), darkeyes (lack of contrast for some users), dynamic background, dynamic facedistance, instability of hand (i.e. camera movement), partial eyecoverage (by eyelids), angle (non-front) capturing, latency,personalized calibration/re-calibration, glasses, contacts-lenses and ofcourse the need to provide results based on real time processing.

Basically, three main challenges have to be resolved in order to detectthe pupil and extract accurate eye measures in real time. Firstly,identifying the eyes, secondly tracking the iris and thirdly trackingthe pupil.

The eye tracking problem deals with detection and localization of an eye(or both eyes) of an observed human. In existing solutions, the exactlocation and the size of the eye typically remain undetermined and abounding box, ellipse, or circle are used to make it. There are existingwidely-used open-source implementations of eye tracking (forexample—http://opencv.org/), however these existing solutions returnsmany false-positive results of the eye locations, therefore degradingthe accuracy of the results.

The iris tracking problem deals with the exact localization of the iris,based on the location of the eye. Since eye localization typically lacksadditional data (such as eye segmentation, corners, etc.) in existingsoftware solutions, and common iris tracking solutions usually involvestwo steps: (a) The first step includes a rough detection of the irislocation, and (b) the second step includes performing an exactlocalization of the iris. This topic is widely covered in theliterature, with many proposed solutions, (e.g.—Deng, Jyh-Yuan, andFeipei Lai. “Region-based template deformation and masking foreye-feature extraction and description.” Pattern recognition 30.3(1997): 403-419). Some of the works are based on edge detection (seeCanny, John. “A computational approach to edge detection.” PatternAnalysis and Machine Intelligence, IEEE Transactions on 6 (1986):679-698), some are model/template based, and some are Hough-transformbased (such as Ballard, Dana H. “Generalizing the Hough transform todetect arbitrary shapes.” Pattern recognition 13.2 (1981): 111-122).

Different prior art references propose different models of iristracking—the simplest model composed of the iris center only (withoutthe radius), the most complex model treats the iris as a generalellipse, and in many other works the iris is modeled as a circle. Mostof the works do not assume special equipment and allow a generic camera.

The pupil tracking problem deals with extraction of the pupil radius andits location. Typically, the problem is solved based on the location andthe radius of the iris (in such case, only the radius of the pupilremains to be detected). The pupil detection problem is a much harderproblem than the iris detection one. This is mainly due to contrastresolution issues at the pupil-iris boundary. However, pupil trackingtechniques have better accuracy since coverage by eyelids is a lesserconcern (except during blinking). Prior art documents that propose asolution to this problem are based on an existing IR illumination inorder to enhance the contrast between the pupil and the iris. Forexample—Mulligan, Jeffrey B. “Image processing for improved eye-trackingaccuracy.” Behavior Research Methods, Instruments, & Computers 29.1(1997): 54-65.

Using the very common IR-based methods, the pupil is detected withoutnecessity in iris detection.

Moreover, no prior art document relates to a method to track the pupilusing a visible-light camera shooting a non-static head.

Although some prior art references seemingly address pupil tracking,they do not extract the pupil radius, and the solved problem is actuallythe iris tracking while the iris center is imposed on the pupil center.

It is therefore an object of the invention to provide a method andsystem for extracting eye markers and deriving capabilities ofmonitoring and improving cognitive states.

It is another object of the invention to provide a method for trackingthe pupil using a visible-light image acquisition device shooting anon-static head.

It is still another object of the invention to extract eye markers froma video stream or file and to detect the pupil.

It is yet another object of the invention to improve eye detectionresults.

Further purposes and advantages of this invention will appear as thedescription proceeds.

SUMMARY OF THE INVENTION

In one aspect the present invention relates to a method for monitoring,testing, and/or improving cognitive abilities of a subject, comprisingthe steps of:

-   -   recording a video of the face and eyes of a subject with an        image acquisition device;    -   detecting the eye's pupil;    -   extracting eye markers from the pupil detected in said video;        and    -   analyzing said extracted eye markers and deriving insights        regarding trends in said subject's cognitive state;

wherein said steps of detecting the pupil and extracting eye markersfrom said video comprise the steps of:

-   -   detecting the eye and the eye region of the image;    -   detecting the iris by receiving as an input said eye region of        said detected image, and providing as an output the iris center        and radius; and    -   detecting and localizing the pupil by receiving as an input said        detected iris center and radius and returning the radius of the        pupil as output.

In an embodiment of the invention, the method further comprises the useof biofeedback, which comprises the steps of:

-   -   interacting with a subject through a computerized application;    -   receiving a derived insights regarding said subject's cognitive        state trends and producing an adapted content for said subject,        which improves said subject's cognitive abilities in “closed        loop”, in accordance with the derived insights about said        subject's cognitive state.

In an embodiment of the invention, the biofeedback is one of thefollowing:

-   -   a. an engaging task;    -   b. a change in subject experience;    -   c. a reduction in game score;    -   d. a slowing down of the game; and    -   e. providing personalized calming content.

In an embodiment of the invention, the image acquisition device is avisible light camera.

In an embodiment of the invention, the step of detecting the pupil andextracting eye markers is done simultaneously for both eyes.

In an embodiment of the invention, in the eye detecting step, a particlefilter is used with the best two particles selected in each iteration,and an eye patch is learnt progressively over multiple frames.

In an embodiment of the invention, the iris detecting step comprises thesteps of:

-   -   detecting local gradients;    -   defining a score to detect a circle located at (x₀, y₀) with a        radius r₀ according to

${Score} = {\int_{\alpha \in {\lbrack{0,{2\pi}}\rbrack}}{{G\left( {{x_{0} + {r_{0}\cos\;\alpha}},{y_{0} + {r_{0}\sin\;\alpha}}} \right)}*\begin{pmatrix}{\cos\mspace{11mu}\alpha} \\{\sin\mspace{11mu}\alpha}\end{pmatrix}d\;\alpha}}$

-   -   Where G(x, y) is the image gradient at point

$\left( {x,y} \right),{{and}\mspace{14mu}\begin{pmatrix}{\cos\mspace{11mu}\alpha} \\{\sin\mspace{11mu}\alpha}\end{pmatrix}}$is the gradient at angle α of said circle;

-   -   defining (x₀, y₀, r₀) which give the highest score as the        detected iris location and radius; and    -   verifying the detected (x₀, y₀, r₀) against a threshold to        determine whether a true iris was detected.

In an embodiment of the invention, if the face is detected but the irisis not, a blink detection is assumed.

In an embodiment of the invention, the step of detecting and localizingthe pupil comprises the steps of:

-   -   detecting the parts of the iris and its area content, which are        occluded by the skin, by checking the angles which do not have        strong gradients;    -   converting the iris image to gray-level;    -   detecting and masking-out the highlights from the surrounding        illumination;    -   computing a 10^(th) percentile intensity value of each radius        and providing a function f(r) which is the 10^(th) percentile        intensity value for each radius r;    -   distinguishing between the lower and the higher parts of said        function f(r) by selecting a value of which results in the        lowest sum of variances of each of the two parts according to:

${r_{0} = {\underset{r}{argmin}\left\lbrack {{\underset{r^{\prime} < r}{Var}\left( {f\left( r^{\prime} \right)} \right)} + {\underset{r^{\prime} > r}{Var}\left( {f\left( r^{\prime} \right)} \right)}} \right\rbrack}};$and

-   -   returning r₀ as the result pupil radius.

All the above and other characteristics and advantages of the inventionwill be further understood through the following illustrative andnon-limitative description of embodiments thereof, with reference to theappended drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically describes the method of pupil detection accordingto an embodiment of the present invention;

FIG. 2 schematically shows an example of the result of the facedetection and eye detection in an image;

FIG. 3 schematically shows an example of the eye region used for irisdetection;

FIG. 4 schematically shows an example of the local gradient found in theeye region;

FIG. 5 schematically shows an example of the response for a certainradius for all the tested iris locations;

FIG. 6 schematically shows the iris parameters that correspond to thepeak in the response function;

FIG. 7 schematically shows an example of a detected iris;

FIG. 8 schematically shows an example of the parts of the iris which areoccluded by the skin and are detected;

FIG. 9 schematically shows an example of a graph of the 10^(th)percentile intensity as a function of the distance from the center ofthe pupil/iris;

FIG. 10 schematically shows an example of the returned pupil radius;

FIG. 11 schematically shows a flow chart of a method according to anembodiments of the present invention; and

FIG. 12 schematically shows a diagram describing the biofeedback of theinvention.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE PRESENT INVENTION

The present invention relates to a system and method for pupil trackingand eye-markers extracting using an image acquisition device such as avisible-light camera shooting a non-static head. In an embodiment of thepresent invention eye-markers are extracted from one eye or from botheyes. The extraction from both eyes uses for averaging the results ofthe two eyes when abnormalities are detected in one of the eyes.

In addition, and as can be seen in FIG. 11, the invention relates to acomputerized application (in a smartphone or a PC) which interacts witha user in step 1101 (for example trough a game or movie, usually in acontext of biofeedback), and takes video shooting of the users face andeyes during said interaction, in step 1102. The method of the inventiontracks pupil and extracts eye markers using a visible-light camerashooting a non-static head. According to the present invention the eyemarkers are extracted in step 1103, such as data about the extractionand contraction of the pupil, the distance between the pupils, and eyemovement. After extracting the eye markers, said extracted eye markersare analyzed in step 1104 and insights are derived regarding thecognitive state of the user. As a result, if necessary, in step 1105, anadapted content is produced for the user, which improves said usercognitive ability in accordance with the derived insights about hiscognitive state.

In one embodiment the eye markers are extracted from a video stream orfile.

In another embodiment, the eye markers are extracted in real-time,directly from the camera.

In an embodiment of the invention, a biofeedback is used in theinvention. FIG. 12 schematically shows a diagram describing thebiofeedback of the invention. A Bio-eye sensor 1023, enables the systemof the invention to utilize eye-biomarkers extracted from the pupil 1022(including but not limited to pupil dilation, eye-movements, and blinks)to monitor the user's cognitive state (including the level of attention,cognitive load and emotional state). Then, if desired, it is possible topresent the user with content-feedback 1024 supporting a desired changein this cognitive state. There are several methods of analysis toextract eye-biomarkers from the eye and pupil signal, used in thepresent invention. For example: (a) taking advantage of the pupil'sdilatory response to cognition-stimulating events. (b) analyzing thefrequency-dynamics of pupil fluctuations in order to determine the stateof mental effort and emotion; and (c) studying eye movements and definestheir normal properties vs abnormalities suggesting possible psychiatricpathology

Examples for improving cognitive state using biofeedback:

-   -   Improve attention: it is possible to immerse the user in an        engaging task (e.g., movie or game) and change user experience        whenever loss of attention is detected (e.g. make movie darker        or slower; alternatively, in context of game-based biofeedback,        it is possible to make change dynamics according to the present        invention: reducing score, slowing game etc.).    -   Improve learning effectiveness: learning material may be tuned,        to maintain an optimal cognitive load (i.e. just at the right        level, not too easy and not too hard, so it is neither boring        nor overwhelming for users).    -   Improve emotional state: stress or emotional distress may be        overcome by providing personalized calming content (visual        and/or auditory) until eye markers indicate relaxation has been        achieved.

FIG. 1. Schematically describes the main steps of the method of pupildetection of the present invention. The method of pupil detection of thepresent invention is divided into four steps: the first step 101 is theEye detection, the second step 102 is the Iris detection andlocalization, the third step 103 is the Blink detection and the fourthstep 104 is the Pupil localization.

The eye detection step 101, is done to determine the rough location ofthe eye, in order to perform the later steps (iris and pupillocalization) in a limited region.

The eye detection step 101 is based on a face detection module wellknown in the art (for example: OpenCV).

First, the image (specifically, its middle part) brightness and contrastare adjusted, and the histogram is equalized. This prepares the imagefor the face detection.

Following, the face detection routine of the face detection module iscalled. If more than one face is detected, the detection is halted. Inthe next step, the eyes are detected using the face detection module'seye detection cascade. The detection is independent of the facedetection. Finally, the eye locations (there can be many “eyes” in theimage) are verified based on the face location and size. Detected eyeswhich are not in the upper half of the detected face are discarded.

However, the eye detection of the face detection module, returns manyfalse-positives of the eye locations. Therefore the present inventionimproves on the existing solution with the following two post-processingmethods which are used to deal with this problem:

1. A particle filter is used with the best two particles selected ineach iteration. A “particle filter” is a well know statistical methodfor removing noise from a signal (Del Moral, Pierre (1996). “Non LinearFiltering: Interacting Particle Solution.” (PDF). Markov Processes andRelated Fields 2 (4): 555-580).

In the case of the present invention pruning of subtle pixel-level noisein the sub-image which covers the eye (i.e. “eye patch”).

Particles are defined as: the detected eye locations (in many frames,more than 2) and the weights are the distance to the locations in thecurrent frame. Using the filter, the returned eye location moves faraway from the previous location only if the new location has a supportover several frames. Furthermore, random false-positives are neglected.

2. Eye patch (the sub-image which covers the eye) is “learnt”(accumulated) over several dozens of frames. In this way, in every framethe eye appearance over the last several seconds is available. The exactlocation of the eye is found using maximal cross-correlation with theknown eye appearance. The output of the eye detector filtered by theparticle filter is used only as a regularizer to the eye patch location.

As a result of these two post-processing steps, the output of the eyedetection module of the present invention becomes more robust (no more“jumps” and exactly two eye locations are returned), more trustable (itis found based on the true eye appearance and not on amachine-learning-based detected), and faster.

FIG. 2 presents a sample result of the eye detection process. First theface of the man 201 is detected and then the eyes 202 are detected.

After the eye detection step 101, the iris detection is a necessary stepfor the pupil detection. The iris is much more prominent in the imagethan the pupil, thus it is much easier to detect. In addition, it can bepractically assumed that the pupil center is the same as the iriscenter. Thus, in order to localize the pupil center, it is needed tosolve the iris detection problem.

The input to this step 102 of iris detection is the eye region of theimage, detected in the previous step of eye detection 101, as shown forexample in FIG. 3 in continuation to the example of FIG. 2.

The purpose of this step 102 is to detect a circular object with an edgeseparating between brighter and darker parts of the image. In otherwords, it is first needed to detect the image gradients which are adirectional changes in the intensity or color in an image.

The gradients are found in the image, based on all three (RGB) channels,as can be seen in FIG. 4, where local gradient are marked in a pictureof an eye.

Next, a score is defined for a circle located at (x₀, y₀) with a radiusr₀:

${Score} = {\int_{\alpha \in {\lbrack{0,{2\pi}}\rbrack}}{{G\left( {{x_{0} + {r_{0}\cos\;\alpha}},{y_{0} + {r_{0}\sin\;\alpha}}} \right)}*\begin{pmatrix}{\cos\;\alpha} \\{\sin\;\alpha}\end{pmatrix}d\;\alpha}}$

Where G(x, y) is the image gradient at point

$\left( {x,y} \right),{{and}\mspace{14mu}\begin{pmatrix}{\cos\;\alpha} \\{\sin\;\alpha}\end{pmatrix}}$is the gradient at angle α of the theoretical circle for which the scoreis computed.

The parameters (x₀, y₀, r₀) which give the highest score are thedetected iris location and radius.

A threshold is defined as:threshold=minScore*strengthFactor

where minScore is a constant with value of 223; and strenthFactor is 1.0if previous iris info still valid, and 1.25 if not.

The score is verified against said threshold to determine whether a trueiris was detected or some random location in an image which does notcontain an iris at all. The latter case (i.e. a false detection of aniris) can happen due to two possible reasons:

1. The eye is closed.

2. The Eye Detection step returned a false positive.

The threshold is derived from the gradient statistics in the image.

In order to verify that the method described above is generally correct,i.e., the detected iris parameters are the true iris location and notdue to a random peak in the response function, the present inventionvisualizes the response function in all three dimensions in the vicinityof the detected parameters. FIG. 5 shows the responses in the x-y planefor each radius.

Each image in FIG. 5 represents the response for a certain radius forall the tested iris center locations. 501 are strong response and 502are weak response. The strong peak in the image corresponding to 20pixel radius suggests that the response is real and not random.

FIG. 6 shows the iris parameters that correspond to the peak in theresponse function. FIG. 7 schematically shows an example of a detectediris.

The verification above was done for many cases, and for all of them theresponse seemed prominent. However, when the search space is expanded,there are cases where there is another local optimum away from the trueiris location, which is better than the response in the true irislocation. Such false positives can be detected using method for eyecenter localizations by means of gradients, and by modifications to thescore calculation (e.g. to limit the weight of strong gradients in orderto reduce the ability of strong edges in the image to pull away from theright solution).

In an embodiment of the invention, when the exact eye location is known,the iris location changes relatively to the eye location only due to eyemovements and then the face movement can be cancelled out.

Due to this improvement, and as the iris movement is small (relative tothe eye location), it can be assumed that the iris location did notchange much from the previous frame. This fact is used to greatlyspeed-up the processing and to make the iris detection more robust. Inthis case the run-time performance is greatly improved. Assuming a videohas “normal” eyes (the eyes are not closed for more than 0.5 seconds,the viewer is looking at the camera, etc.), the frames are processedabout 6-8 times faster than in the case where iris location changes fromframe to frame.

However, if during several frames it is not detected that the iris nearthe previously known iris location, a full search in the eye region isperformed.

In an embodiment of the invention blinks are detected. A non-detectediris is a strong indicator of a blink (the iris is not visible), and thepresent invention treats a non-detected iris (assuming the face isdetected) as a blink detection. In one embodiment of the method of thepresent invention, the blink detection algorithm relies completely onthe quality of the iris detection. However, in another embodiment of theinvention a separate blink detection algorithm is based on skindetection, machine learning, verification against neighboring frames orother methods.

The last step of the method of pupil detection in the present inventionis the pupil localization step 104 (FIG. 1). The pupil is a darkercircle located inside a larger circle (the iris), which has a relativelyconstant color. The circles have a common center. In order todistinguish between the pupil and the iris, and find the pupil radius,the pixels of the pupil and the iris should contain statistically(considering the noise) different values. The possibility to detect thepupil radius strongly depends on the specific video—its resolution,noise, illumination, and on the iris color of the analyzed eye. Brightiris colors are easier than dark ones.

The input to this step 104 of pupil localization, are the irisparameters (center and radius), whose detection is described in the step102 of iris detection.

The purpose of step 104 is to find the radius of the pupil, as itscenter is identical to the center of the iris.

First, the parts of the iris that are occluded by the skin are detected,by checking the angles that do not have strong gradients; i.e. assumingcolor and intensity representation of the eyelid in an image issignificantly different than that of the iris, the algorithm findsangles along which moderate change in color and intensity conveyposition of eyelid relative to the iris. These heuristics aredemonstrated by the two circles 801 and 802 in FIG. 8. Part of the irisis visible 801, and part is hidden under the eyelid 802.

Then, the following steps are performed:

1. Convert the iris to gray-level.

2. Detect and mask-out the highlights from the surrounding illumination.

3. Compute the 10^(th) percentile intensity value of each radius.Generally speaking, using the 10th percentile implies using the top 10%according to a sort based on some measure (in this case, the radius ofthe pupil). According to the present invention a 1 dimensional vector ofintensities is formed, starting from the center of the pupil (which isdarkest) and moving out to towards the iris (which is expected to be abit brighter at least at outskirts). The border of the pupil radius isdefined using the start point of the 10th percentile in grey-levelintensity.

The result is a 1D function, f(r), where r is the radius and f(r) is the10^(th) percentile intensity. The function should return lowerintensities for small values of r (the pupil) and higher intensities forlarge values of r (the iris). FIG. 9 schematically shows an example of agraph of the 10^(th) percentile intensity as a function of the distancefrom the center of the pupil/iris (i.e. the radius).

As a final step, the method of the present invention distinguishesbetween the lower and the higher parts of the function by selecting avalue r₀ of which results in the lowest sum of variances of each of thetwo parts, and as can be seen in FIG. 10.

The r₀ is calculated according to:

$r_{0} = {\underset{r}{argmin}\left\lbrack {{\underset{r^{\prime} < r}{Var}\left( {f\left( r^{\prime} \right)} \right)} + {\underset{r^{\prime} > r}{Var}\left( {f\left( r^{\prime} \right)} \right)}} \right\rbrack}$

r₀ is the returned pupil radius.

In an embodiment of the invention, instead of collapsing the pupilinformation to 1D function, all pupil pixels are analyzed.

At first, a mask that removes the high-lighted and skin pixels isapplied. Following, the pupil radius is computed. This is the radius rthat minimizes the following:

${\lambda{\sum\limits_{x \in {{Pupil}{(r)}}}\left( {x - \overset{\sim}{x}} \right)^{2}}} + {\left( {1 - \lambda} \right){\sum\limits_{y \notin {{Pupil}{(r)}}}\left( {y - \overset{\sim}{y}} \right)^{2}}}$

where λ is a weighting factor, which is set to 0.7, and x and y arepixel intensity values.

The confidence score of the pupil with a given radius r is:

$s = \frac{{\overset{\sim}{x} - \overset{\sim}{y}}}{\sqrt{{{var}(x)} + {{var}(y)}}}$

Which is normalized to a [0,1] range, so that s=1 becomes 0.1 and s=2becomes 0.9 (these values were empirically found as “bad” and “good”confidence):

$s^{\prime} = \frac{1}{1 + e^{{- {({s + 1.5})}} \cdot 4.395}}$

The confidence score is the system output, and it is used in theaveraging post-processing step of the pupil values of the two eyes(pupil with higher confidence has a higher weight in the averaging).

The usual recording distance from the phone camera is about 30 cm. Thediameter of the human iris, is about 11-12 mm.

Generally, for cameras with a standard field of view, such as SGS3 oriPhone 4:

${{pixel\_ per}{\_ mm}} \cong {100\frac{\sqrt{{image\_ resolution}{\_ in}{\_ MP}}}{{plane\_ distance}{\_ in}{\_ cm}}}$

Thus, the diameter of the iris in an image is about 40 pixels accordingto the following calculation:

${\frac{100\sqrt{1.280 \cdot 0.720}}{30} \cdot 12} \cong 40.$

In an embodiment of the invention, the motion blur problem is addressedby taking advantage of multiple frames from the video rather thanworking frame by frame. As data is repeated and the underlying data inhigh frequency sampling is constant, while the motion blur is anadditive noise, it can significantly improve/remove it.

Another problem with which the present invention deals is the glassesproblem. Glasses may affect the detection for several reasons:

1. The detected object (eyes, iris or pupil) may be occluded by theglasses frame.

2. The glasses lens may introduce geometric distortion, lower contrastor color tint.

3. Stains on the glasses may interfere with the image of the detectedobject.

In an embodiment of the invention, in case where the user has glasses oreye contact an auto compensate for glasses and contact lensesidentification is operated, and a geometric correction (as if using acomplementary lens) may be applied. In another embodiment of theinvention, the method of the invention manages with reduced field ofview (when eyes are partially shut) to an extent, as interference withview are expected to be minimal, as significant interference willdisturb the user.

In an embodiment of the invention, optimal quality stills photos aretaken in parallel to video, with max resolution, smallest aperture(if/when dynamic), optimal ISO (probably lowest possible, higher if itis must for poor lighting conditions), optimized shutter speed (slowestpossible depending on stability indications by accelerometer, trial anderror, etc.), spot metering mode (to pupil), optimized white-balancing,optimize/enhance dynamic range. The present invention uses stills photosto improve/correct accuracy and to compensate lighting configuration todetermine actual lighting condition.

In an embodiment of the invention, lighting conditions/changes aremeasured/tracked through pixel brightness in different eye sections andcompensate for changes. Alternatively, average brightness of all pixelsin images recorded by the front\back camera may indicate such changes inlighting conditions.

In an embodiment of the invention, the distance is normalized bymeasuring change in face size (use constant features regardless ofexpressions). A standard feature/measure is the distance between theeyes, However, the present invention can use other features such as facewidth at eyes, etc.

In an embodiment of the invention, head/face orientation is normalized,including compensation for elliptical iris/pupil.

Due to the nature of the eye, many parameters change only a littlebetween adjacent frames and this fact can be used for a more stablesolution:

-   -   For the eye detection stage: the eye shape and its rough        location.    -   For the iris detection stage: radius, rough location, and color.    -   For the pupil radius detection stage: The pupil radius.

In an embodiment of the invention, the iris detection algorithm can beimproved by using information other than the iris circle: the eye shape,the eye color. Furthermore, model-based methods, such as RANSAC andHough transform (which are common feature extraction techniques used inimage analysis to identify imperfect instances of geometric shapes (inour case a circle or ellipse)), have to be considered.

In an embodiment of the invention, other problems the invention dealswith are:

Accuracy: A dynamic model (e.g., particle filter) and super-resolutiontechniques can be used through multiple consecutive frames to obtainsub-pixel accuracy. Also, occasional high quality still pictures can betaken to further improve and tune the accuracy.

Dynamic lighting: the (front) camera brightness can becontrolled/optimized to improve the dynamic range of extracted objects(specifically, eyes and pupil).

Dark eyes: the ‘red’ colors of the spectrum can be filtered, and thismode can be used as an approximation of the IR camera (including IRspectrum if/when not filtered by the camera).

Dynamic background: using eye detection methods described above, allredundant background can be filtered out.

Personalized calibration: in the embodiment of the present invention,the system is calibrated for current user and settings, and is switchedto tracking mode (see below). In case of tracking loss, the systemperforms a fresh acquisition (and re-calibration), and when ready, itswitches back to tracking mode.

Latency & Real-time: algorithms and performance are optimized to providefastest (minimal latency—milliseconds) extraction and delivery of theextracted measures. In cases of heavy processing (e.g., re-calibration)or insufficient processing resources, a reduced frame rate may be usedto maintain real-time delivery.

Although embodiments of the invention have been described by way ofillustration, it will be understood that the invention may be carriedout with many variations, modifications, and adaptations, withoutexceeding the scope of the claims.

The invention claimed is:
 1. A method for monitoring, testing, and/orimproving cognitive abilities of a subject, comprising steps of: a.recording a video of a face and eyes of a subject with an imageacquisition device; b. detecting an eye's pupil; c. extracting eyemarkers from the pupil detected in said video; and d. analyzing saidextracted eye markers and deriving insights regarding trends in saidsubject's cognitive state; the method further comprising use ofbiofeedback, which comprises steps of: interacting with the subjectthrough a computerized application; receiving derived insights regardingsaid subject's cognitive state trends from the extracted eye markers,and producing an adapted content for said subject, which improves saidsubject's cognitive abilities in a closed loop, in accordance with thederived insights about said subject's cognitive state; wherein saidsteps of detecting the pupil and extracting eye markers from said videocomprise steps of: I. detecting an eye and an eye region from an imageof the face and the eyes of the subject; II. detecting an iris byreceiving as an input said eye region of said detected image, andproviding as an output an iris center and radius; and III. detecting andlocalizing the pupil by receiving as an input said iris center andradius and returning a radius of the pupil as output.
 2. The methodaccording to claim 1, wherein the biofeedback is one of the following:a. a change in subject experience of an engaging task; b. a reduction ina game score; c. a slowing down of a game; and d. a provision ofpersonalized calming content.
 3. The method according to claim 1,wherein the image acquisition device is a visible light camera.
 4. Themethod according to claim 1, wherein, the step of detecting the pupiland extracting the eye markers is done simultaneously for both eyes. 5.The method according to claim 1, wherein if the face is detected but theiris is not, a blink detection is assumed.
 6. The method according toclaim 1, wherein, in the eye detecting step, a particle filter is usedand two best particles are selected in each iteration, and an eye patchis learnt progressively over multiple frames.
 7. A method formonitoring, testing, and/or improving cognitive abilities of a subject,comprising steps of: recording a video of a face and eyes of a subjectwith an image acquisition device; detecting an eye's pupil; extractingeye markers from the pupil detected in said video; and analyzing saidextracted eye markers and deriving insights regarding trends in saidsubject's cognitive state; wherein said steps of detecting the pupil andextracting eye markers from said video comprise steps of: I. detectingan eye and an eye region from an image of the face and the eyes of thesubject: II. detecting an iris by receiving as an input said eye regionof said detected image, and providing as an output an iris center andradius; and III. detecting and localizing the pupil by receiving as aninput said iris center and radius and returning a radius of the pupil asoutput; wherein the iris detecting step comprises steps of: a. detectinglocal gradients; b. defining a score to detect a circle located at (x₀,y₀) with a radius r₀ according to${Score} = {\int_{\alpha \in {\lbrack{0,{2\pi}}\rbrack}}{{G\left( {{x_{0} + {r_{0}\cos\;\alpha}},{y_{0} + {r_{0}\sin\;\alpha}}} \right)}*\begin{pmatrix}{\cos\;\alpha} \\{\sin\;\alpha}\end{pmatrix}d\;\alpha}}$ where G(x, y) is an image gradient at point$\left( {x,y} \right),{{and}\mspace{14mu}\begin{pmatrix}{\cos\;\alpha} \\{\sin\;\alpha}\end{pmatrix}}$ is a gradient at angle α of said circle; c. defining(x₀, y₀, r₀) which give a highest score as a detected iris location andradius; and d. verifying the defined (x₀, y₀, r₀) against a threshold todetermine whether a true iris was detected.
 8. The method according toclaim 7, wherein the image acquisition device is a visible light camera.9. The method according to claim 7, wherein the step of detecting thepupil and extracting the eye markers is done simultaneously for botheyes.
 10. The method according to claim 7, wherein, in the eye detectingstep, a particle filter is used and two best particles are selected ineach iteration, and an eye patch is learnt progressively over multipleframes.
 11. The method according to claim 7, wherein if the face isdetected but the iris is not, a blink detection is assumed.
 12. A methodfor monitoring, testing, and/or improving cognitive abilities of asubject, comprising steps of: recording a video of a face and eyes of asubject with an image acquisition device; detecting an eye's pupil;extracting eye markers from the pupil detected in said video; andanalyzing said extracted eye markers and deriving insights regardingtrends in said subject's cognitive state; wherein said steps ofdetecting the pupil and extracting eye markers from said video comprisesteps of: I. detecting an eye and an eye region from an image of theface and the eyes of the subject: II. detecting an iris by receiving asan input said eye region of said detected image, and providing as anoutput an iris center and radius; and III. detecting and localizing thepupil by receiving as an input said iris center and radius and returninga radius of the pupil as output, wherein the step of detecting andlocalizing the pupil comprises steps of: a. detecting parts of the irisand its area content, which are occluded by skin, by checking angleswhich do not have strong gradients; b. converting an image of the iristo gray-level; c. detecting and masking-out highlights from surroundingillumination; d. computing a 10^(th) percentile intensity value of eachradius and providing a function f(r) which is a 10^(th) percentileintensity value for each radius r; e. distinguishing between a lowerpart and a higher part of said function f(r) by selecting a value ofwhich results in a lowest sum of variances of each of the two lower andhigher parts according to:${r_{0} = {\underset{r}{argmin}\left\lbrack {{\underset{r^{\prime} < r}{Var}\left( {f\left( r^{\prime} \right)} \right)} + {\underset{r^{\prime} > r}{Var}\left( {f\left( r^{\prime} \right)} \right)}} \right\rbrack}};$and f. returning r₀ as a pupil radius.
 13. The method according to claim12, wherein the image acquisition device is a visible light camera. 14.The method according to claim 12, wherein the step of detecting thepupil and extracting the eye markers is done simultaneously for botheyes.
 15. The method according to claim 12, wherein, in the eyedetecting step, a particle filter is used and two best particles areselected in each iteration, and an eye patch is learnt progressivelyover multiple frames.
 16. The method according to claim 12, wherein ifthe face is detected but the iris is not, a blink detection is assumed.